Skip to content

fix: generate complete padded ngrams for short sequences - #805

Open
xiaoyao-666-coder wants to merge 1 commit into
NaturalNode:masterfrom
xiaoyao-666-coder:fix/ngram-short-sequence-padding
Open

xiaoyao-666-coder wants to merge 1 commit into
NaturalNode:masterfrom
xiaoyao-666-coder:fix/ngram-short-sequence-padding

Conversation

@xiaoyao-666-coder

@xiaoyao-666-coder xiaoyao-666-coder commented Nov 13, 2026 •

Copy link
Copy Markdown

Proposed Changes

NGrams.trigrams('hello', '[start]', '[end]') currently returns four arrays, including ['[start]', 'hello'] and ['hello', '[end]'], which are not trigrams. It also misses ['[start]', 'hello', '[end]']. NGramsZH has the same problem. Independently generating the two padded ends assumes that the input has at least n - 1 tokens.

Pad the sequence with n - 1 copies of each requested boundary symbol before taking complete windows. A shared internal helper keeps word and Chinese n-grams consistent, preserves caller arrays, and leaves unpadded inputs alone. Existing statistics now count the complete windows as well.

The one-word example now produces:

[
  ['[start]', '[start]', 'hello'],
  ['[start]', 'hello', '[end]'],
  ['hello', '[end]', '[end]']
]

Companion documentation for the gh-pages branch: #806.

Validation

On Windows with Node.js 24.22.1:

  • Added 18 Jasmine specs for both implementations, including short strings/arrays, one-sided padding, empty inputs, empty-string boundary symbols, unigrams, array preservation, and frequency statistics. Against unmodified master, 16 fail; all pass with this change.
  • All 45 n-gram specs pass.
  • npm run build:tests and npm test pass: 916 specs, zero failures.
  • npm run lint, npm run build:esm, and npm run smoke:esm pass.
  • Standard JS passes for the changed JavaScript files, with two existing object-shorthand warnings in the statistics return object. A focused jscpd scan reports no clones.
  • git diff --check passes.

The database-backed I/O suite was not run locally.

Checklist before merging

  • Added Jasmine tests
  • Reviewed TypeScript declarations; no public signature changes

Prepared with Codex assistance.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant